Nonlinear Optimization

Given a single function f that depends on one or more independent variable, we want to find the values of those variables where f is maximized or minimized. Often the computational cost is dominated by the the cost of evaluating f (and also perhaps its partial derivatives with respect to all variables). Finding a global extremum is, in general, a very difficult problem. Two standard heuristics are widely used: i) find local extrema starting from widely varying values of the independent variables, and then pick the most extreme of these; ii) perturb a local extremum by taking a finite amplitude step away from it, and then see if your routine can get to a better point, or “always” to the same one. Recently, “simulated annealing” methods have demonstrated important successes on a variety of global optimization problems. The diagram below describes a function in an interval [a, b]. The derivative vanishes at the points B,C,D,E, F . The points B and D are local but not global maxima. The points C and E are local but not global minima. The global maximum occurs at F where the derivative vanishes. The global minimum occurs at the left endpoint A of the interval so that the derivative need not vanish.